{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Hyperparameter Tuning with Grid Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise you will train a base model and then try to find better combinations of hyperparameter values using the grid search technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the cells below to read prepared data on the [Invesco QQQ Trust, Series 1 (NASDAQ: QQQ)](https://finance.yahoo.com/quote/QQQ/) ETF from 1999 to 2017. We have already engineered some technical indicators as features and cleaned the data. The DataFrame also includes the raw level of the VIX (Volatility Index)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fut_ret_5d_is_pos</th>\n",
       "      <th>volume_adi</th>\n",
       "      <th>volume_obv</th>\n",
       "      <th>volume_cmf</th>\n",
       "      <th>volume_fi</th>\n",
       "      <th>volume_em</th>\n",
       "      <th>volume_sma_em</th>\n",
       "      <th>volume_vpt</th>\n",
       "      <th>volume_vwap</th>\n",
       "      <th>volume_mfi</th>\n",
       "      <th>...</th>\n",
       "      <th>momentum_ppo_signal</th>\n",
       "      <th>momentum_ppo_hist</th>\n",
       "      <th>momentum_pvo</th>\n",
       "      <th>momentum_pvo_signal</th>\n",
       "      <th>momentum_pvo_hist</th>\n",
       "      <th>momentum_kama</th>\n",
       "      <th>others_dr</th>\n",
       "      <th>others_dlr</th>\n",
       "      <th>others_cr</th>\n",
       "      <th>vix_close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4.676000e+03</td>\n",
       "      <td>4.676000e+03</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4.676000e+03</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4.676000e+03</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "      <td>4676.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.563302</td>\n",
       "      <td>5.699828e+09</td>\n",
       "      <td>5.732778e+09</td>\n",
       "      <td>0.064547</td>\n",
       "      <td>-2.194298e+06</td>\n",
       "      <td>-0.075677</td>\n",
       "      <td>-0.078392</td>\n",
       "      <td>-5.095351e+07</td>\n",
       "      <td>58.064601</td>\n",
       "      <td>51.933904</td>\n",
       "      <td>...</td>\n",
       "      <td>0.133332</td>\n",
       "      <td>0.000744</td>\n",
       "      <td>-0.482135</td>\n",
       "      <td>-0.484135</td>\n",
       "      <td>0.002000</td>\n",
       "      <td>58.302141</td>\n",
       "      <td>0.039945</td>\n",
       "      <td>0.024127</td>\n",
       "      <td>17.604880</td>\n",
       "      <td>20.094363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.496030</td>\n",
       "      <td>5.161116e+09</td>\n",
       "      <td>3.516850e+09</td>\n",
       "      <td>0.159343</td>\n",
       "      <td>2.349555e+07</td>\n",
       "      <td>6.905866</td>\n",
       "      <td>2.014736</td>\n",
       "      <td>6.960159e+07</td>\n",
       "      <td>30.826903</td>\n",
       "      <td>16.820708</td>\n",
       "      <td>...</td>\n",
       "      <td>1.762790</td>\n",
       "      <td>0.518646</td>\n",
       "      <td>7.860636</td>\n",
       "      <td>6.560185</td>\n",
       "      <td>3.872669</td>\n",
       "      <td>30.940742</td>\n",
       "      <td>1.781342</td>\n",
       "      <td>1.777656</td>\n",
       "      <td>62.775450</td>\n",
       "      <td>8.598108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>-4.938774e+08</td>\n",
       "      <td>-1.463058e+08</td>\n",
       "      <td>-0.407576</td>\n",
       "      <td>-2.747434e+08</td>\n",
       "      <td>-148.776736</td>\n",
       "      <td>-39.307413</td>\n",
       "      <td>-1.798704e+08</td>\n",
       "      <td>18.765972</td>\n",
       "      <td>4.400570</td>\n",
       "      <td>...</td>\n",
       "      <td>-8.335029</td>\n",
       "      <td>-2.952249</td>\n",
       "      <td>-31.336335</td>\n",
       "      <td>-25.045400</td>\n",
       "      <td>-13.991364</td>\n",
       "      <td>19.295283</td>\n",
       "      <td>-8.953572</td>\n",
       "      <td>-9.380061</td>\n",
       "      <td>-63.836136</td>\n",
       "      <td>9.140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.301191e+08</td>\n",
       "      <td>1.889184e+09</td>\n",
       "      <td>-0.046602</td>\n",
       "      <td>-8.508723e+06</td>\n",
       "      <td>-0.266021</td>\n",
       "      <td>-0.077014</td>\n",
       "      <td>-1.220200e+08</td>\n",
       "      <td>34.768331</td>\n",
       "      <td>40.201831</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.517182</td>\n",
       "      <td>-0.270265</td>\n",
       "      <td>-5.518779</td>\n",
       "      <td>-4.739690</td>\n",
       "      <td>-2.454129</td>\n",
       "      <td>34.997522</td>\n",
       "      <td>-0.683252</td>\n",
       "      <td>-0.685597</td>\n",
       "      <td>-29.799202</td>\n",
       "      <td>13.840000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.178166e+09</td>\n",
       "      <td>7.129624e+09</td>\n",
       "      <td>0.060795</td>\n",
       "      <td>1.477789e+06</td>\n",
       "      <td>0.028058</td>\n",
       "      <td>0.020283</td>\n",
       "      <td>-3.354444e+07</td>\n",
       "      <td>45.596244</td>\n",
       "      <td>51.787090</td>\n",
       "      <td>...</td>\n",
       "      <td>0.439187</td>\n",
       "      <td>-0.004300</td>\n",
       "      <td>-0.774752</td>\n",
       "      <td>-0.710966</td>\n",
       "      <td>-0.179074</td>\n",
       "      <td>45.650949</td>\n",
       "      <td>0.097418</td>\n",
       "      <td>0.097371</td>\n",
       "      <td>-8.072255</td>\n",
       "      <td>18.195001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.084870e+10</td>\n",
       "      <td>9.032024e+09</td>\n",
       "      <td>0.174096</td>\n",
       "      <td>8.161734e+06</td>\n",
       "      <td>0.369925</td>\n",
       "      <td>0.131703</td>\n",
       "      <td>2.171185e+07</td>\n",
       "      <td>79.038435</td>\n",
       "      <td>63.497209</td>\n",
       "      <td>...</td>\n",
       "      <td>1.156162</td>\n",
       "      <td>0.268306</td>\n",
       "      <td>4.172172</td>\n",
       "      <td>3.744792</td>\n",
       "      <td>2.316098</td>\n",
       "      <td>80.297697</td>\n",
       "      <td>0.795889</td>\n",
       "      <td>0.792739</td>\n",
       "      <td>59.697190</td>\n",
       "      <td>23.842500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.491344e+10</td>\n",
       "      <td>1.006329e+10</td>\n",
       "      <td>0.722899</td>\n",
       "      <td>1.322961e+08</td>\n",
       "      <td>82.926192</td>\n",
       "      <td>15.099535</td>\n",
       "      <td>4.575173e+07</td>\n",
       "      <td>149.820914</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>5.413036</td>\n",
       "      <td>3.955131</td>\n",
       "      <td>34.734169</td>\n",
       "      <td>23.164445</td>\n",
       "      <td>24.304876</td>\n",
       "      <td>150.070245</td>\n",
       "      <td>16.859905</td>\n",
       "      <td>15.580564</td>\n",
       "      <td>208.999637</td>\n",
       "      <td>80.860001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 88 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       fut_ret_5d_is_pos    volume_adi    volume_obv   volume_cmf  \\\n",
       "count        4676.000000  4.676000e+03  4.676000e+03  4676.000000   \n",
       "mean            0.563302  5.699828e+09  5.732778e+09     0.064547   \n",
       "std             0.496030  5.161116e+09  3.516850e+09     0.159343   \n",
       "min             0.000000 -4.938774e+08 -1.463058e+08    -0.407576   \n",
       "25%             0.000000  8.301191e+08  1.889184e+09    -0.046602   \n",
       "50%             1.000000  4.178166e+09  7.129624e+09     0.060795   \n",
       "75%             1.000000  1.084870e+10  9.032024e+09     0.174096   \n",
       "max             1.000000  1.491344e+10  1.006329e+10     0.722899   \n",
       "\n",
       "          volume_fi    volume_em  volume_sma_em    volume_vpt  volume_vwap  \\\n",
       "count  4.676000e+03  4676.000000    4676.000000  4.676000e+03  4676.000000   \n",
       "mean  -2.194298e+06    -0.075677      -0.078392 -5.095351e+07    58.064601   \n",
       "std    2.349555e+07     6.905866       2.014736  6.960159e+07    30.826903   \n",
       "min   -2.747434e+08  -148.776736     -39.307413 -1.798704e+08    18.765972   \n",
       "25%   -8.508723e+06    -0.266021      -0.077014 -1.220200e+08    34.768331   \n",
       "50%    1.477789e+06     0.028058       0.020283 -3.354444e+07    45.596244   \n",
       "75%    8.161734e+06     0.369925       0.131703  2.171185e+07    79.038435   \n",
       "max    1.322961e+08    82.926192      15.099535  4.575173e+07   149.820914   \n",
       "\n",
       "        volume_mfi  ...  momentum_ppo_signal  momentum_ppo_hist  momentum_pvo  \\\n",
       "count  4676.000000  ...          4676.000000        4676.000000   4676.000000   \n",
       "mean     51.933904  ...             0.133332           0.000744     -0.482135   \n",
       "std      16.820708  ...             1.762790           0.518646      7.860636   \n",
       "min       4.400570  ...            -8.335029          -2.952249    -31.336335   \n",
       "25%      40.201831  ...            -0.517182          -0.270265     -5.518779   \n",
       "50%      51.787090  ...             0.439187          -0.004300     -0.774752   \n",
       "75%      63.497209  ...             1.156162           0.268306      4.172172   \n",
       "max     100.000000  ...             5.413036           3.955131     34.734169   \n",
       "\n",
       "       momentum_pvo_signal  momentum_pvo_hist  momentum_kama    others_dr  \\\n",
       "count          4676.000000        4676.000000    4676.000000  4676.000000   \n",
       "mean             -0.484135           0.002000      58.302141     0.039945   \n",
       "std               6.560185           3.872669      30.940742     1.781342   \n",
       "min             -25.045400         -13.991364      19.295283    -8.953572   \n",
       "25%              -4.739690          -2.454129      34.997522    -0.683252   \n",
       "50%              -0.710966          -0.179074      45.650949     0.097418   \n",
       "75%               3.744792           2.316098      80.297697     0.795889   \n",
       "max              23.164445          24.304876     150.070245    16.859905   \n",
       "\n",
       "        others_dlr    others_cr    vix_close  \n",
       "count  4676.000000  4676.000000  4676.000000  \n",
       "mean      0.024127    17.604880    20.094363  \n",
       "std       1.777656    62.775450     8.598108  \n",
       "min      -9.380061   -63.836136     9.140000  \n",
       "25%      -0.685597   -29.799202    13.840000  \n",
       "50%       0.097371    -8.072255    18.195001  \n",
       "75%       0.792739    59.697190    23.842500  \n",
       "max      15.580564   208.999637    80.860001  \n",
       "\n",
       "[8 rows x 88 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# DO NOT MODIFY - load data and display basic statistics\n",
    "df = pd.read_csv(\"Data.csv\")\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'd like to try and predict the direction of 5-day future returns. Run the cell below to split the data and prepare for model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - Define features and target and split data\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = df.drop(columns=[\"fut_ret_5d_is_pos\", \"Date\"])\n",
    "y = df[\"fut_ret_5d_is_pos\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a `RandomForestClassifier` and train it using its default hyperparameter values. As this is a tree-based model, you do not need to scale the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomForestClassifier(n_jobs=-1, random_state=52)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;RandomForestClassifier<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestClassifier.html\">?<span>Documentation for RandomForestClassifier</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>RandomForestClassifier(n_jobs=-1, random_state=52)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "RandomForestClassifier(n_jobs=-1, random_state=52)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# FILL IN - Instantiate a RandomForestClassifier and fit it to the training data\n",
    "# Use random_state=52 for reproducibility\n",
    "# Set n_jobs=-1 to enable parallel processing using all available CPU cores\n",
    "clf = RandomForestClassifier(n_jobs=-1, random_state=52)\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will focus on precision as our performance metric, as we would like to avoid False Positives as much as possible.  \n",
    "Below, we have provided a function that plots 5-fold cross-validated precision scores. Study it and invoke it to plot the learning curves using the training set. You should be able to observe that the model is overfitting to the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/farid/.pyenv/versions/3.10.6/envs/udacity/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, f\"{metric.capitalize()} is\", len(result))\n",
      "/home/farid/.pyenv/versions/3.10.6/envs/udacity/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, f\"{metric.capitalize()} is\", len(result))\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 34784 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "# DO NOT MODIFY - plotter function\n",
    "def plot_learning_curve(model, X, y, cv=5, n_jobs=-1):\n",
    "    train_sizes, train_scores, test_scores = learning_curve(\n",
    "        model,\n",
    "        X,\n",
    "        y,\n",
    "        cv=cv,\n",
    "        n_jobs=n_jobs,\n",
    "        scoring=\"precision\",\n",
    "    )\n",
    "    train_scores_mean = np.mean(train_scores, axis=1)\n",
    "    test_scores_mean = np.mean(test_scores, axis=1)\n",
    "    plt.plot(train_sizes, train_scores_mean, label=\"CV training score\")\n",
    "    plt.plot(train_sizes, test_scores_mean, label=\"CV test score\")\n",
    "    plt.title(\"Learning curve for Random Forest Classifier\")\n",
    "    plt.xlabel(\"Training examples\")\n",
    "    plt.ylabel(\"Precision\")\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "# FILL IN - Plot the learning curve for the RandomForestClassifier using the training data\n",
    "plot_learning_curve(clf, X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What was the average cross-validated precision score on the training set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.4660277780001759)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# FILL IN - Get the 5-fold cross-validated precision of the classifer on the training data\n",
    "# Use n_jobs=-1 for parallel processing\n",
    "precision = cross_val_score(clf, X_train, y_train, cv=5, n_jobs=-1, scoring=\"precision\")\n",
    "precision.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And how does this score compare to the precision on the test set? - **HINT:** Use the fitted classifier's `predict()` method to get an array of predictions on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.6216216216216216)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.metrics import precision_score\n",
    "\n",
    "# FILL IN - Get the precision of the classifier on the test data\n",
    "y_pred = clf.predict(X_test)\n",
    "precision_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that you can use the `get_params()` method of the classifierto see a list of its hyperparameters and other settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'class_weight': None,\n",
       " 'criterion': 'gini',\n",
       " 'max_depth': None,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 2,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'monotonic_cst': None,\n",
       " 'n_estimators': 100,\n",
       " 'n_jobs': -1,\n",
       " 'oob_score': False,\n",
       " 'random_state': 52,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we have picked 3 different values for each of the 4 major hyperparameters of `RandomForestClassifier`. Using Scikit-Learn's `GridSearchCV` class, perform a 5-fold cross-validated grid search using the provided search grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-3 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-3 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-3 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-3 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-3 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-3 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=5, estimator=RandomForestClassifier(n_jobs=-1, random_state=52),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;max_depth&#x27;: [5, 10, 20],\n",
       "                         &#x27;min_samples_leaf&#x27;: [1, 2, 4],\n",
       "                         &#x27;min_samples_split&#x27;: [2, 7, 15],\n",
       "                         &#x27;n_estimators&#x27;: [50, 100, 200]},\n",
       "             scoring=&#x27;precision&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;GridSearchCV<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.model_selection.GridSearchCV.html\">?<span>Documentation for GridSearchCV</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>GridSearchCV(cv=5, estimator=RandomForestClassifier(n_jobs=-1, random_state=52),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;max_depth&#x27;: [5, 10, 20],\n",
       "                         &#x27;min_samples_leaf&#x27;: [1, 2, 4],\n",
       "                         &#x27;min_samples_split&#x27;: [2, 7, 15],\n",
       "                         &#x27;n_estimators&#x27;: [50, 100, 200]},\n",
       "             scoring=&#x27;precision&#x27;)</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">best_estimator_: RandomForestClassifier</label><div class=\"sk-toggleable__content fitted\"><pre>RandomForestClassifier(max_depth=5, min_samples_leaf=2, min_samples_split=7,\n",
       "                       n_estimators=200, n_jobs=-1, random_state=52)</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RandomForestClassifier<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestClassifier.html\">?<span>Documentation for RandomForestClassifier</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RandomForestClassifier(max_depth=5, min_samples_leaf=2, min_samples_split=7,\n",
       "                       n_estimators=200, n_jobs=-1, random_state=52)</pre></div> </div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=5, estimator=RandomForestClassifier(n_jobs=-1, random_state=52),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'max_depth': [5, 10, 20],\n",
       "                         'min_samples_leaf': [1, 2, 4],\n",
       "                         'min_samples_split': [2, 7, 15],\n",
       "                         'n_estimators': [50, 100, 200]},\n",
       "             scoring='precision')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# DO NOT MODIFY - the `hyperspace` of hyperparameters to search\n",
    "search_grid = {\n",
    "    'n_estimators': [50, 100, 200],\n",
    "    'max_depth': [5, 10, 20],\n",
    "    'min_samples_split': [2, 7, 15],\n",
    "    'min_samples_leaf': [1, 2, 4]\n",
    "}\n",
    "\n",
    "# FILL IN - Instantiate a GridSearchCV object with the fitted RandomForestClassifier, the search grid, 5-fold cross-validation, and precision scoring.\n",
    "# Fit it to the training data\n",
    "# Don't forget to set n_jobs=-1 for parallel processing. This may take a minute or two even with parallel processing.\n",
    "search = GridSearchCV(clf, param_grid=search_grid, n_jobs=-1, cv=5, scoring='precision')\n",
    "search.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store the best parameters, best score, and best estimator (model). (These are attributes of `search`.) Feel free to print out the best CV precision score. How does it compare to the base model? Which combination of values yielded this result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Get the best parameters, best score, and best estimator from the GridSearchCV object\n",
    "best_params = search.best_params_\n",
    "best_score = search.best_score_\n",
    "best_estimator = search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.5708863641184567)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 5,\n",
       " 'min_samples_leaf': 2,\n",
       " 'min_samples_split': 7,\n",
       " 'n_estimators': 200}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to see the top 5 results in detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_max_depth</th>\n",
       "      <th>param_min_samples_leaf</th>\n",
       "      <th>param_min_samples_split</th>\n",
       "      <th>param_n_estimators</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1.562755</td>\n",
       "      <td>0.074617</td>\n",
       "      <td>0.152285</td>\n",
       "      <td>0.087112</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>200</td>\n",
       "      <td>{'max_depth': 5, 'min_samples_leaf': 2, 'min_s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.355263</td>\n",
       "      <td>0.441739</td>\n",
       "      <td>0.506627</td>\n",
       "      <td>0.550802</td>\n",
       "      <td>0.570886</td>\n",
       "      <td>0.224430</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>1.597074</td>\n",
       "      <td>0.123847</td>\n",
       "      <td>0.226982</td>\n",
       "      <td>0.065240</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>200</td>\n",
       "      <td>{'max_depth': 5, 'min_samples_leaf': 4, 'min_s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.355263</td>\n",
       "      <td>0.440972</td>\n",
       "      <td>0.506627</td>\n",
       "      <td>0.550802</td>\n",
       "      <td>0.570733</td>\n",
       "      <td>0.224518</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1.507539</td>\n",
       "      <td>0.089787</td>\n",
       "      <td>0.234889</td>\n",
       "      <td>0.143421</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>200</td>\n",
       "      <td>{'max_depth': 5, 'min_samples_leaf': 4, 'min_s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.355263</td>\n",
       "      <td>0.440972</td>\n",
       "      <td>0.506627</td>\n",
       "      <td>0.550802</td>\n",
       "      <td>0.570733</td>\n",
       "      <td>0.224518</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1.682186</td>\n",
       "      <td>0.055887</td>\n",
       "      <td>0.146827</td>\n",
       "      <td>0.053028</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>200</td>\n",
       "      <td>{'max_depth': 5, 'min_samples_leaf': 2, 'min_s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.346801</td>\n",
       "      <td>0.436300</td>\n",
       "      <td>0.505170</td>\n",
       "      <td>0.550802</td>\n",
       "      <td>0.567815</td>\n",
       "      <td>0.226778</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>1.472832</td>\n",
       "      <td>0.156795</td>\n",
       "      <td>0.227551</td>\n",
       "      <td>0.086435</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>200</td>\n",
       "      <td>{'max_depth': 5, 'min_samples_leaf': 4, 'min_s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.343333</td>\n",
       "      <td>0.440208</td>\n",
       "      <td>0.504451</td>\n",
       "      <td>0.550802</td>\n",
       "      <td>0.567759</td>\n",
       "      <td>0.227052</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "14       1.562755      0.074617         0.152285        0.087112   \n",
       "23       1.597074      0.123847         0.226982        0.065240   \n",
       "20       1.507539      0.089787         0.234889        0.143421   \n",
       "17       1.682186      0.055887         0.146827        0.053028   \n",
       "26       1.472832      0.156795         0.227551        0.086435   \n",
       "\n",
       "    param_max_depth  param_min_samples_leaf  param_min_samples_split  \\\n",
       "14                5                       2                        7   \n",
       "23                5                       4                        7   \n",
       "20                5                       4                        2   \n",
       "17                5                       2                       15   \n",
       "26                5                       4                       15   \n",
       "\n",
       "    param_n_estimators                                             params  \\\n",
       "14                 200  {'max_depth': 5, 'min_samples_leaf': 2, 'min_s...   \n",
       "23                 200  {'max_depth': 5, 'min_samples_leaf': 4, 'min_s...   \n",
       "20                 200  {'max_depth': 5, 'min_samples_leaf': 4, 'min_s...   \n",
       "17                 200  {'max_depth': 5, 'min_samples_leaf': 2, 'min_s...   \n",
       "26                 200  {'max_depth': 5, 'min_samples_leaf': 4, 'min_s...   \n",
       "\n",
       "    split0_test_score  split1_test_score  split2_test_score  \\\n",
       "14                1.0           0.355263           0.441739   \n",
       "23                1.0           0.355263           0.440972   \n",
       "20                1.0           0.355263           0.440972   \n",
       "17                1.0           0.346801           0.436300   \n",
       "26                1.0           0.343333           0.440208   \n",
       "\n",
       "    split3_test_score  split4_test_score  mean_test_score  std_test_score  \\\n",
       "14           0.506627           0.550802         0.570886        0.224430   \n",
       "23           0.506627           0.550802         0.570733        0.224518   \n",
       "20           0.506627           0.550802         0.570733        0.224518   \n",
       "17           0.505170           0.550802         0.567815        0.226778   \n",
       "26           0.504451           0.550802         0.567759        0.227052   \n",
       "\n",
       "    rank_test_score  \n",
       "14                1  \n",
       "23                2  \n",
       "20                2  \n",
       "17                4  \n",
       "26                5  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_results = pd.DataFrame(search.cv_results_)\n",
    "search_results.sort_values('rank_test_score').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-use the same `plot_learning_curve()` function we provided earlier to plot the learning curve for the best estimator found using training data. How does it compare to the learning curve of the original classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/farid/.pyenv/versions/3.10.6/envs/udacity/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, f\"{metric.capitalize()} is\", len(result))\n",
      "/home/farid/.pyenv/versions/3.10.6/envs/udacity/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, f\"{metric.capitalize()} is\", len(result))\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 45268 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# FILL IN - Plot the learning curve for the best estimator using the training data\n",
    "plot_learning_curve(best_estimator, X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, use this estimator to evaluate the test set and get the new test performance score. How does it compare?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.6578947368421053)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# FILL IN - Get the precision of the best model on test data\n",
    "y_pred = best_estimator.predict(X_test)\n",
    "precision_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Get predictions on the training and test sets\n",
    "y_pred_train = clf.predict(X_train)\n",
    "y_pred_test = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 22100 characters]",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "def plot_confusion_matrix(y_true, y_pred):\n",
    "    conf_mat = confusion_matrix(y_true, y_pred)\n",
    "    sns.heatmap(conf_mat, annot=True, fmt=\"d\", cmap=\"Blues\")\n",
    "    plt.xlabel(\"Predicted Labels\")\n",
    "    plt.ylabel(\"Actual Labels\")\n",
    "    plt.title(\"Confusion Matrix\")\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_confusion_matrix(y_test, y_pred_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[347,  14],\n",
       "       [552,  23]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Plot confusion matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_test, y_pred_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
